-
Notifications
You must be signed in to change notification settings - Fork 1k
[stable2509] Backport #9912 #10044
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[stable2509] Backport #9912 #10044
Conversation
Please cherry-pick the changes locally and resolve any conflicts. git fetch origin backport-9912-to-stable2509
git worktree add --checkout .worktree/backport-9912-to-stable2509 backport-9912-to-stable2509
cd .worktree/backport-9912-to-stable2509
git reset --hard HEAD^
git cherry-pick -x b21cbb58ab50d5d10371393967537f6f221bb92f
git push --force-with-lease |
Fixes gossiping and scalability issues in the statement-store networking. 1. Reduced gossiping traffic by propagating only recent statements instead of all. 2. Added an early check for statements that the node already has to skip duplicate processing. 3. Added splitting of large statement batches to stay under MAX_STATEMENT_NOTIFICATION_SIZE; oversized individual statements are skipped. 4. MAX_STATEMENT_NOTIFICATION_SIZE was updated to the commonly used 1MB, which drastically improved the gossiping speed. 5. Notifications are sent asynchronously. I don't see much difference in performance, but according to @lexnv, it's better to do: #9296. 6. Added a 10s timeout to handle very slow or disconnected peers. Internal optimizations to the gossip protocol. No downstream changes required. Related PR: #9965 - After this PR, nodes don't send all statements to new peers anymore, only the recent ones. - After restarting, the node doesn't re-gossip statements it wasn't gossiped. - Broadcasting notifications to all peers when the first peer is slow is limited. We could instead use a FuturesUnordered. --------- Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Bastian Köcher <[email protected]> (cherry picked from commit b21cbb5)
058a7e5
to
d6703dc
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The backport conflicted because it has been merged in the opposite order of #10040 I think.
But all good and fixed.
This pull request is amending an existing release. Please proceed with extreme caution,
Emergency Bypass
If you really need to bypass this check: add |
Backport #9912 into
stable2509
from AndreiEres.See the documentation on how to use this bot.